Learning hierarchical policies from human ratings

Robots are on the verge of becoming ubiquitous. In the form of affordable humanoid toy robots, autonomous cars, vacuum robots or quadrocopters, robots are becoming part of our everyday life. As of today, most of these robots still follow largely hard coded behavior routines. Constraining a robot’s behavior to pre-programmed routines, however, limits its potential in several important ways. For example, programming even simple behavior patterns is a challenging task and programming behavior with human like performance by hand seems impossible. The goal of this thesis, thus, is to develop methods which allow robots to learn solutions to tasks through trial and error instead of relying on manual programming. These learned solutions should fulfill a range of desired properties. Foremost, the solutions should be learned on real world robots and not be constrained to simplified simulation environments. Furthermore, we would like the robot to learn versatile solutions which are able to cope with different variations of a task. Finally, we would like to be able to also solve more complicated tasks which require sequencing of multiple skills. In the first part of this thesis, we propose an algorithm to learn such versatile solutions. The proposed algorithm aims to find a hierarchical policy, consisting of a gating policy and a set of sub-policies. The gating policy selects a sub-policy and the sub-policy decides which action to take. Each of the learned sub-policies may be able to solve the task or a part of the overall task. To learn multiple sub-policies from one sample set, we employ an expectation-maximization based learning algorithm, where the sub-policies are updated according to their responsibilities for individual samples. These responsibilities indicate how likely it is that a certain sub-policy generated a given state-action pair. By constraining our learning algorithm to solutions which increase the entropy of the responsibilities, the robot will learn a set of sub-policies that encode different solutions for the same task. This kind of concurrency is highly desirable as it allows the robot to learn back-up solutions which may still be valid even if the original solution fails due to changes in the robot or environment. In the second part of this thesis, we tackle the challenge of learning skills directly from stateaction trajectories. A common approach in robot learning is to assume access to some sort of parametrized skill to represent the robot’s behavior over multiple time steps. These skills are usually either movement primitives (MPs) or parametrized feedback controllers. While especially MPs have been a cornerstone in advancing the state of the art in robot learning, it is not clear how to learn from the actions taken throughout the execution of a skill and MPs usually do not encode feedback. In the discrete state-action reinforcement learning (RL) setting, macro-actions, or options, have been introduced to address the problem of learning temporally correlated actions, which can be viewed as a form of skill. To connect robot learning with the advances in the field of discrete state-action RL, we propose a probabilistic framework to infer options from stateaction trajectory observations. The inference is based on a hidden Markov model (HMM), where the options indices are modeled as latent variables and where inference can be performed by adapting well known expectation-maximization algorithms. Because this framework allows for the inference of parametric policies, it is also compatible with policy search (PS) methods, a class of RL algorithms which is at the core of many recent successes in robot learning. Learning methods, such as the one we propose in the first chapter of this thesis, enable robots to learn from trial and error instead of having to program specific solutions. Unfortunately, the